Predicting DXL Memory and CPU Usage

Mathias Mamsch - Fri Oct 18 05:58:47 EDT 2013

Just my two cents: If you are running one script on a large set of modules, how about a parallelization framework? In python for example it is cool, using the multiprocessing library you can do something like this:

def doWork (item):
        # expensive work
        pass

with multiprocessing.pool(5) as pool: # use 5 parallel processes
     result = pool.map(doWork, [ ... list of items ... ]

The multiprocessing library will take care about starting 5 processes, splitting up the list of items into parts, feeding it to the 5 processes, executing the function doWork on the items and sending the results back to the calling process. I figure something equal can be done in DXL. It will not only speed up your process (due to parallelization -> most operations seem to be CPU bound), but also get rid of memory issues, since you can use MORE memory even on DOORS 8 (because of parallel processes) and memory will get freed by the external processes after completion. You will however require a couple of licenses (at least one more than usual) on parallel processing, but I guess this is something most companies might be willing to take for the runtime of the scripts.

If you are interested in doing something like this together, msg me. However regarding your original question we implemented successfully to measure the DOORS memory allocation using an external tool from DXL and safely stopping the script at a configurable memory barrier. This might also a way to go. When it comes to actually predicting the memory usage, this is kind of unreliable. Depending on the size of data you will encounter next you will waste more or less memory, so you probably cannot say reliably, how many modules you will be able to execute without actually doing it ...

Just my two cents, regards, Mathias

VinayN - Mon Oct 21 04:57:09 EDT 2013

Usage of memory management perms:

1) initNotDisplayedOpenModules() - This is 1st perm which records modules already opened before any operation begins and should be called at start of any looping operation.

2) DisplayMemoryWarning() - 2nd perm which checks memory usage of application and closes modules opened in back end when application consumes more memory than the prescribed warning level and this should be placed inside a for or while loop so that memory is monitored in each iteration in regular interval.
3) closeNotDisplayedOpenedModules() - 3rd perm which does all cleanup operation at last and closees all modules opened in back end during end of any operation

Usage of these three perms is shown below in the example code and these 3 DXL perms should be used in combination and when user is looping through all objects in a module with huge data or objects.

/********************************************/

   Object o = current
               initNotDisplayedOpenModules()    // First perm which records modules opened before any operation
               Module m = current
               int i = 0
               for o in m do
               {
                   if (DisplayMemoryWarning())   //second perm which checks memory usage and closes                        //modules opened during any operation
                   {
                              break
                   }
                   i = o."Absolute Number"
                   gotoObject(i, m)
                   o = next o
               }
               closeNotDisplayedOpenedModules() // Third perm which does all cleanup operation and tries to close //all modules which were opened during the operation

/********************************************/

1) initNotDisplayedOpenModules() is used before the start of the operation
2) DisplayMemoryWarning()) is used during actual operation i.e. with in for or while loop which monitors memory used by application.
3) closeNotDisplayedOpenedModules() is used at the end of the operation.

Rational DOORS DXL developer should use these 3 perms in any custom DXL code where it involves looping of all objects in a big module with huge data and links to other module. This should be used inside a loop which actually loops through all objects to collect data.

Customizing these new memory management perms:

Memory usage varies from system to system. Some system may have good memory and thus DOORS can use more memory for its operations. Since we cannot fix a value or generalize the maximum memory that can be used by Rational DOORS, we have given some registry values to customize it. This helps Rational DOORS DXL developer to set the system memory that DOORS can use.

Default values set in Rational DOORS:

By default there are certain parameters set and these can be overridden by registry entries set under DOORS client registry under config key. Below are the registry string keys with its importance and usage

MEM_LEVEL_CLOSE - Its default value is 1000 MB and upon reaching this level, the memory management perm will start closing opened modules that it keeps track of. Customer can override this max value by having a string key registry entry MEM_LEVEL_CLOSE under DOORS Client->Config and specify suitable value. Customer should know what is the maximum MB of memory that his system can allocate to DOORS and give that value under this registry key. For example if a machine can allocate 2 GB of memory to doors.exe without making it crash then value for this registry key would be 1800 or 2000. So they have to decide the value by trial and error method. Higher the value good performance but machine should have such high memory to allocate.

MEM_LEVEL_WARN - Its default value is 800 MB and upon reaching this level, the memory management perm will warn user to either continue or abort the operation to avoid crash. Customer can override this by having a registry entry MEM_LEVEL_WARN under DOORS Cleint->Config and specify suitable value. Significance of this value is - Rational DOORS will start closing all back end open modules when application reaches MEM_LEVEL_CLOSE mb of memory and even after closing all background modules if memory consumed by Rational DOORS is more than the value specified by this MEM_LEVEL_WARN registry entry then Rational DOORS shows a warning message to user. So it means Rational DOORS is consuming more memory than mentioned warning level by customer. So an ideal value would be 800 to 1600 for a 2 GB Ram machine.

MEM_LEVEL_STEP - Its default value is 200 MB and this value is used to step up the MEM_LEVEL_WARN level at run time. A warning will be displayed at run time and at that time you can step up the MEM_LEVEL_WARN level by looking at the system memory. Customer can override this by having a registry entry MEM_LEVEL_STEP under DOORS Client->Config and specify suitable value.

Note: This is available from DOORS 9.3.0.7 onwards. A manual interpretation may be needed (after DOORS 9.5) when using these API's (perms) because a warning will be displayed when Rational DOORS touches MEM_LEVEL_WARN level and thus user has to select either to continue or abort. Also there is no much side effect of using these perms while looping through objects but it may affect performance to very negligible extent because we are doing memory checks at regular intervals and trying to close module at regular intervals and customer have to adjust these settings via registry entries so that performance is not effected and also crash does not occur.

Mathias Mamsch - Fri Oct 18 05:58:47 EDT 2013

def doWork (item):
        # expensive work
        pass

with multiprocessing.pool(5) as pool: # use 5 parallel processes
     result = pool.map(doWork, [ ... list of items ... ]

Just my two cents, regards, Mathias

ProgrammingRai - Wed Sep 21 08:20:29 EDT 2016

Hello Mathias,

how does this parallelized work with modules in doors work?

Thanks,

Rainer